Guild icon
Project Sekai
🔒 UIUCTF 2023 / ❌-web-futuredisk
Avatar
futuredisk - 500 points
Category: Web Description: I'm from the year 2123. Here's what I did:
  • Mounted my 10 exabyte flash drive
  • fallocate -l 8E haystack.bin
  • dd if=flag.txt bs=1 seek=[REDACTED] conv=notrunc of=haystack.bin
  • gzip haystack.bin
  • Put haystack.bin.gz on my web server for you to download HTTP over Time Travel is a bit slow, so I hope gzipping it made it a little faster to download :) https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz Files: No files.
Tags: No tags.
Sutx pinned a message to this channel. 06/30/2023 5:15 PM
Avatar
@DreyAnd wants to collaborate 🤝
Avatar
@jayden wants to collaborate 🤝
Avatar
@jayden any idea what do we do with this huge file lmao
17:31
can't even download it
17:31
i guess HTTP over Time Travel is a hint
Avatar
might be something to do with the range header https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range
The Range HTTP request header indicates the part of a document that the server should return. Several parts can be requested with one Range header at once, and the server may send back these ranges in a multipart document. If the server sends back ranges, it uses the 206 Partial Content for the response. If the ranges are invalid, the server ret...
17:45
though dont know where it is in the 8eb file cause seek=[REDACTED]
17:46
that's a cool idea
17:46
use range to get parts
Avatar
@unpickled admin bot wants to collaborate 🤝
Avatar
@Violin wants to collaborate 🤝
Avatar
@ElleuchX1 wants to collaborate 🤝
Avatar
anyone looking at this one rn?
Avatar
@irogir wants to collaborate 🤝
Avatar
from the idea jayden gave, i was looking into looping through all ranges with Range: <byte_start>:<byte_end>
05:16
which works
05:16
but it would only be luck if we manage to match a flag in there early
05:16
it would take months probably to loop through this file
05:17
i think the right question to ask is how did they even host this file
05:17
no way they have servers with so much storage
05:17
so must be some trick with dummy files
Avatar
does the server time out for you too on a head request?
Avatar
nope
05:19
works for me
05:19
05:19
but yea, range works
Avatar
are you able to tell if flag is after or before some offset?
Avatar
i guess no because dd if=flag.txt bs=1 seek=[REDACTED] conv=notrunc of=haystack.bin
05:42
seek is redacted
Avatar
does the command mean flag is at offset X
05:43
out of the whole big offset of 10 exabyte or whatever (edited)
Avatar
Yeah
Avatar
are all other bytes 0?
Avatar
hmm not exactly
05:45
saw some other chall like this, try this: import requests current_offset = 0 SESS = requests.Session() while True: chunk = SESS.get("https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": "bytes={}-{}".format(current_offset, current_offset+2048)}).content print(chunk)
Avatar
how big is whole file btw, in offset
05:46
the max offset
Avatar
the haystack.bin is 8 exabytes
Avatar
how big is that
05:47
in order of 2
05:47
8*10**18
05:48
somehow we need a binary search where each query can tell us if flag is in half range, not sure if feasible, otherwise there are trick besides range offset?
Avatar
yeah i'm not sure exactly, though i think you are going in a good direction
05:53
binary search is a very good idea in fact
Avatar
gztool maybe help
Avatar
someone try chatgpt
05:59
in fact does anyone know the diff between chal2 and this, both 8 exabytes
Avatar
might be a long shot, but maybe we can play with the server date somehow?
Avatar
they said chal2 is a little worse but i dont see diff
Avatar
Avatar
DreyAnd
might be a long shot, but maybe we can play with the server date somehow?
the challenge implies it's coming from the future
Avatar
maybe server diff
06:00
would prob be helpful to see what 2 has on top of 1
06:00
which might be to patch 1 intended
Avatar
let me take a look
06:03
server response headers and file sizes are exact the same
Avatar
ugh we can be first if we solve this one
06:36
but it's so annoying
Avatar
its gzip misc i guess, idk what is the web part
07:57
can you open a modmail ticket and ask if we on right track by using range offset and not missing out some other web aspect detail?
Avatar
i can sure, idk if they'll like that question tho
Avatar
you can also format whatever you think is proper
Avatar
Yeah ill try
Avatar
im the rev chal they did sanity check our progress
Avatar
Alright sent a message, let's see
Avatar
Avatar
DreyAnd
hmm not exactly
btw this is just gzip file signature, so my bad
08:03
though there is other stuff except null bytes in other chunks
Avatar
example?
Avatar
would be hard to do if there are other non null stuff
Avatar
nvm my bad, it was the same signature being returned on every request
Avatar
all others null byte?
Avatar
Yeah
08:06
i guess let's try binary search
08:06
with range header
08:07
match anything that isn't a null byte or file signature
Avatar
there's also this weird UUUUU... stuff
08:18
import requests target_signature = b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\x03\xec\xc1\x01\x01\x00\x00\x00\x80\x90\xfe\xaf\xee\x08\n" current_offset = 0 chunk_size = 2048 while True: response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={current_offset}-{current_offset + chunk_size - 1}"} ) chunk = response.content if b"\x00" not in chunk and target_signature not in chunk: print("Matched data found in chunk starting at offset", current_offset) print(chunk) current_offset += chunk_size
Avatar
got ignored on modmail btw @sahuang 😂
Avatar
Avatar
DreyAnd
there's also this weird UUUUU... stuff
where is this
08:41
everywhere
08:42
starting from offset 8192-10239
Avatar
Avatar
DreyAnd
got ignored on modmail btw @sahuang 😂
ping them
08:50
i think we are just trying to understand if this is rabbit hole
08:50
and web stuff is needed for other parts
08:51
if not, we can dig into this range query and see what we can do with gzip
Avatar
sent another one
Avatar
ill ask after you get some info
Avatar
Avatar
sahuang
i think we are just trying to understand if this is rabbit hole
but yeah mostly asking this
Avatar
i am getting the ✅ so the message has arrived
08:55
ah ok
08:55
Sorry, we’ve pinged the challenge author - think they are currently asleep. Will let you know when they are awake
Avatar
i see
10:26
are we able to sanity check if range query is not a rabbit hole? or confirm gzip thing is the path (more like a sanity check than a hint)
Avatar
that's what i asked, i guess also why he said he's hesitant to give hints
10:28
gonna get back to this challenge now tho
Avatar
Avatar
DreyAnd
so must be some trick with dummy files
Hosting a file of that size (8 exabytes) on a server is indeed not feasible due to current limitations in storage capacity. In this scenario, the description mentions that the file haystack.bin was created using the fallocate command, which effectively preallocates the space on the disk for the file but does not actually write any data to it. This means that the file is essentially empty, and its size is only limited by the available disk space. By using this technique, the challenge creator can simulate a very large file without actually consuming significant storage resources. It's important to note that since the file is empty, downloading it will not take up excessive bandwidth or storage on the user's side.
Avatar
that would explain it
11:15
It's important to note that since the file is empty, downloading it will not take up excessive bandwidth or storage on the user's side. how tf do they think i should download it
Avatar
So header starts with b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\x03\xec\xc1\x01\x01\x00\x00\x00\x80\x90\xfe\xaf\xee\x08\n\x00...
11:15
Can you get the tailer?
Avatar
what's the tailer
11:16
last 8 bytes
11:16
yeah
11:16
you can
Avatar
i assume you can, given the total size and offset
11:16
can you get it
11:16
i believe the chal is gzip forensics or whatsoever
Avatar
i guess we can do something like Range: bytes=9223372036854775799-9223372036854775807 to get the last 8 bytes
11:19
kind of sucks this CTF had only 2 real web challenges
Avatar
Avatar
DreyAnd
i guess we can do something like Range: bytes=9223372036854775799-9223372036854775807 to get the last 8 bytes
all null bytes, interesting
Avatar
since 9223372036854775807 is the whole length
11:21
maybe the goal is to get it decompressed on server and download that instead?
Avatar
that could be a part too
11:23
since it looks like a forensics challenge
11:23
but misses a web part
Avatar
also i think we need to understand whats the difference between part 1 and 2
11:23
i have no idea
11:23
on web part
11:23
files are the same it seems
Avatar
trying binary search rn
11:24
and next ill look into that
Avatar
file headers are the same
Avatar
Avatar
DreyAnd
trying binary search rn
how do you binsearch?
Avatar
well i tried* i guess that failed already XD
11:25
can't split such file in half
11:25
since yet again too big
Avatar
Avatar
sahuang
file headers are the same
let me loop through both files with my script and see if they're the same
Avatar
Avatar
DreyAnd
import requests target_signature = b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\x03\xec\xc1\x01\x01\x00\x00\x00\x80\x90\xfe\xaf\xee\x08\n" current_offset = 0 chunk_size = 2048 while True: response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={current_offset}-{current_offset + chunk_size - 1}"} ) chunk = response.content if b"\x00" not in chunk and target_signature not in chunk: print("Matched data found in chunk starting at offset", current_offset) print(chunk) current_offset += chunk_size
this script?
Avatar
yeah just comment out the if's
11:27
chunks look different
11:27
which ones?
11:28
header are same
11:28
for chall 1 and chall 2
Avatar
hmm ic
11:29
what are these data lol
11:29
btw do you know how to recreate the file locally?
11:29
i wonder if gzip has "uiuc..." embedded in it, or just flag.txt
Avatar
lemme give that a try
11:30
let's at least see how it's encoded
11:30
on a smaller size file
Avatar
when flag is passed into it
Avatar
Avatar
sahuang
i wonder if gzip has "uiuc..." embedded in it, or just flag.txt
Yeah it won't have the flag inside the gzip
11:36
it will have a reference to the .bin file
11:36
so no way to match that
Avatar
can you share a sample
11:36
file with flag.txt in it
11:38
oh actually flag.txt is input, actual .bin wont have flag.txt, it just has the flag content encoded
Avatar
do this
11:39
create a flag.txt with uiuctf{random}
11:39
fallocate -l 10mb example.bin
11:39
dd if=flag.txt bs=1 seek=1024 conv=notrunc of=example.bin
11:39
you can see example.bin has the flag here
11:40
pipe to xxd if you need to check the offset
11:40
and then gzip example.bin
Avatar
but wait, maybe gzip has a different format where it embeds example.bin at the top or end
11:42
and you can just get that
11:42
and then binwalk it
Avatar
let me try a larger file
11:42
1G for example
11:43
for me it's on the top
Avatar
it is, but i wonder with larger offset and file
Avatar
lmk what happens with a bigger file
11:43
yeye
Avatar
seek is byte?
11:44
1024 is 1mb?
Avatar
you can put anything there, doesn't matter at this point
11:44
since that's where the flag will be placed inside the example.bin
11:45
we first need to extract the .bin
11:45
seek is just the offset
Avatar
yeah its at the front
11:48
interesting
Avatar
how much does it take tho?
11:49
im being afraid if the original bin is 8 exabytes, it may take as much at the top XD
11:49
or hopefully less
Avatar
Avatar
DreyAnd
im being afraid if the original bin is 8 exabytes, it may take as much at the top XD
wdym?
Avatar
like the first 8 exabytes will be the bin file
Avatar
the smaller seek is, the faster its done
Avatar
why haystack.bin isnt at their file's top lol
Avatar
that's what is buzzing me rn
Avatar
When named pipes are not enough: Provide arbitrarily large files of pseudo-random test data, without consuming disk space. - GitHub - s5k6/otffs: When named pipes are not enough: Provide arbitraril...
11:52
they prob used this
11:52
sth like this
11:53
that can "provide a file system providing arbitrarily large (well, almost. Exabytes is the current limit) files of generated content"
Avatar
the provided files are seekable, and multiple reads from the same offset of the same file are guaranteed to return identical data.
11:54
hmm
Avatar
gonna go out, back in a few hours
12:09
need to refresh my head
12:09
wish i spent time on actual webs lol, now all i have is a headache
Avatar
so any web guy find out whats the actual difference between chal 1 and 2
Avatar
any forensics guy hahah
15:54
chunks are different, so has to be be a different trick
Avatar
Avatar
DreyAnd
Click to see attachment 🖼️
.
Avatar
Strellic — Today at 4:13 PM 💀 this seems like range header this seems like compression forensics which is fake web presumably you can read the gzip header or something and maybe it tells you where the flag is
Avatar
tried already
16:30
maybe not hard enough
16:30
😂
Avatar
yeah i think this seems the way
16:30
cuz
16:30
many solves
16:31
probably chal2 has the structure such that this doesnt work
Avatar
gonna try in a little
Avatar
@kanon wants to collaborate 🤝
Avatar
Avatar
DreyAnd
import requests target_signature = b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\x03\xec\xc1\x01\x01\x00\x00\x00\x80\x90\xfe\xaf\xee\x08\n" current_offset = 0 chunk_size = 2048 while True: response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={current_offset}-{current_offset + chunk_size - 1}"} ) chunk = response.content if b"\x00" not in chunk and target_signature not in chunk: print("Matched data found in chunk starting at offset", current_offset) print(chunk) current_offset += chunk_size
@unpickled admin bot here
17:10
b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\x03\xec\xc1\x01\x01\x00\x00\x00\x80\x90\xfe\xaf\xee\x08\n" if you query first few bytes you get this
17:11
actually we should read https://www.ietf.org/rfc/rfc1952.txt
Avatar
indeed, i'm just coming home
17:14
how much time is left
Avatar
24 hrs
Avatar
unpickled admin bot 07/01/2023 5:14 PM
helloo
Avatar
Avatar
sahuang
actually we should read https://www.ietf.org/rfc/rfc1952.txt
unpickled admin bot 07/01/2023 5:14 PM
did you mean this?
Avatar
so there are a few things
17:14
yeah
Avatar
unpickled admin bot 07/01/2023 5:14 PM
alr ty
Avatar
read from 2.3. Member format Each member has the following structure: +---+---+---+---+---+---+---+---+---+---+ |ID1|ID2|CM |FLG| MTIME |XFL|OS | (more-->) +---+---+---+---+---+---+---+---+---+---+
17:15
ID1 ID2 are fixed
17:15
CM = 8 denotes the "deflate"
17:15
FLG is 0
17:16
If FTEXT is set, the file is probably ASCII text
Avatar
Avatar
sahuang
read from 2.3. Member format Each member has the following structure: +---+---+---+---+---+---+---+---+---+---+ |ID1|ID2|CM |FLG| MTIME |XFL|OS | (more-->) +---+---+---+---+---+---+---+---+---+---+
unpickled admin bot 07/01/2023 5:16 PM
thats only at the begining though?
Avatar
yeah im reading the beginning
Avatar
unpickled admin bot 07/01/2023 5:16 PM
did anyone try the Digest header alr?
17:16
or no?
Avatar
what is that
17:16
last 8 bytes are all null, thats another weird thing i said
Avatar
Avatar
sahuang
what is that
unpickled admin bot 07/01/2023 5:17 PM
ill just check, but sometimes on download webservers will send a digest of the data beign downloaded
Avatar
oh i see why filename isnt in beginning of gz file
Avatar
unpickled admin bot 07/01/2023 5:20 PM
there is a etag
Avatar
Avatar
DreyAnd
Click to see attachment 🖼️
example shows 1f 8b 08 08 they set it as 1f 8b 08 00 The last byte is FLG This flag byte is divided into individual bits as follows: bit 0 FTEXT bit 1 FHCRC bit 2 FEXTRA bit 3 FNAME
If FNAME is set, an original file name is present, terminated by a zero byte.
17:20
They set all bits to 0, so no filename
17:21
Now, XFL is set to 02 our example set as 00
XFL = 2 - compressor used maximum compression, slowest algorithm
17:21
so the first 10 bytes are clear
Avatar
unpickled admin bot 07/01/2023 5:24 PM
is it possible to use etags with range?
17:24
or will the etag always be of the whole file
Avatar
always same
17:25
also same for chal 1 and 2
17:27
weird thing is trailer is all null
Avatar
Avatar
sahuang
Click to see attachment 🖼️
it's supposed to contain crc-32 checksum
17:29
wait no
17:29
it ends with many null bytes but has sth before that
17:29
ending prob doesnt matter
17:29
U...UUUUUUUUUUUUUUUUUU\xa5=8 \x00\x00\x00\x00\x10\xd2\xff\xd5\x1dA\x00\x00\x00\x00\x00...
17:30
this is the ending part
17:30
anything not U or \x00 are useful i think
Avatar
Avatar
sahuang
anything not U or \x00 are useful i think
yup ^^
Avatar
even with the RFC i'm not getting any more ideas lol
17:47
like yeah, it explains some things but no idea tf i should do
Avatar
yeah it explained a few things, and that header doesnt have flag info, but filename isnt present
17:48
rfc mentioned "trailer" but i didnt see the spec
Avatar
I tried replacing UUU...U with a new line, but it's the same thing over and over? Idk how this helps
17:59
can you write that to a file
17:59
and check hex dump
Avatar
Avatar
kanon
I tried replacing UUU...U with a new line, but it's the same thing over and over? Idk how this helps
unpickled admin bot 07/01/2023 6:00 PM
ye because its a primarily empty .bin, alot of things zip to the same thing
Avatar
'\xc3\x84\xc5\xb8\xc3\x89\x03\x01\x00\x00\x00\x00 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\x00\x00\x00\x00\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\x00\x00\x00\x00\x00\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\x00\x00\x00\x00\x00\xc2\xbb\xcb\x87\xc2\xb5\x11T' '\xc3\xbf\xc3\x89\x03\x01\x00\x00\x00\x00 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\x00\x00\x00\x00\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\x00\x00\x00\x00\x00\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\x00\x00\x00\x00\x00\xc2\xbb\xcb\x87\xc2\xb5\x11T' '\xc3\xbf\xc3\x89\x03\x01\x00\x00\x00\x00 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\x00\x00\x00\x00\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\x00\x00\x00\x00\x00\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\x00\x00\x00\x00\x00\xc2\xbb\xcb\x87\xc2\xb5\x11T' It's the same except for the very beginning If it's empty, it seems like a simpler repetition, idk
Avatar
interesting
Avatar
unpickled admin bot 07/01/2023 6:13 PM
ye the hard part tho is prob going to be figuring out where its different
18:14
because in theory it should be different where the flag is
Avatar
are you able to recreate a file locally with same header?
Avatar
anyone got a spare k8s cluster😂
18:15
let's just distribute the looping process and see
Avatar
unpickled admin bot 07/01/2023 6:15 PM
(presumably the same header with a shorter file) (edited)
Avatar
Avatar
unpickled admin bot
(presumably the same header with a shorter file) (edited)
no
Avatar
unpickled admin bot 07/01/2023 6:15 PM
maybe slightly diff
Avatar
bits are different in FLG
Avatar
Avatar
sahuang
example shows 1f 8b 08 08 they set it as 1f 8b 08 00 The last byte is FLG This flag byte is divided into individual bits as follows: bit 0 FTEXT bit 1 FHCRC bit 2 FEXTRA bit 3 FNAME
If FNAME is set, an original file name is present, terminated by a zero byte.
here
Avatar
Avatar
sahuang
no
unpickled admin bot 07/01/2023 6:16 PM
i meant a not 10 exabyte file lmao
18:16
cuz he said k8s cluster
Avatar
ah ok
Avatar
Avatar
sahuang
here
In that case, what about the null bytes on the way? Delete it?
Avatar
yea they're irrelevant
Avatar
b'\x1f\xc3\xa3\x08\x02\x03\xc3\x8f\xc2\xa1\x01\x01\xc3\x84\xc3\xaa\xcb\x9b\xc3\x98\xc3\x93\x08\n\xc3\x84\xc5\xb8\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T\xc3\xbf\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T\xc3\xbf\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T' the range is bytes=0-100000 and replace "U" and "\x00", but the headers are different?
Avatar
unpickled admin bot 07/01/2023 6:31 PM
ye
18:31
the header has actual content
Avatar
Avatar
kanon
b'\x1f\xc3\xa3\x08\x02\x03\xc3\x8f\xc2\xa1\x01\x01\xc3\x84\xc3\xaa\xcb\x9b\xc3\x98\xc3\x93\x08\n\xc3\x84\xc5\xb8\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T\xc3\xbf\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T\xc3\xbf\xc3\x89\x03\x01 \xcb\x87\xe2\x97\x8aFPa\x0f\x0e\x04\xc3\x84\xc2\xb8_\x1bA\xc3\x96=8\x10\xc3\x9a\x7fm\x04\x15\xcb\x86\xe2\x80\xa1@\xc2\xbb\xcb\x87\xc2\xb5\x11T' the range is bytes=0-100000 and replace "U" and "\x00", but the headers are different?
unpickled admin bot 07/01/2023 6:31 PM
whats weird
18:31
is
18:31
i did this locally
18:31
and its not that seq of bytes repeating
18:31
its just \x00s (edited)
18:33
heres my local gzipped version ig
Avatar
download locally, remove \x00 and U and its repeating those bytes (edited)
18:36
did that for me too
Avatar
import requests chunk = requests.get("https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range":"bytes=0-100000"}).content out_file = open("out.bin", "wb") chunk = chunk.replace(b"UU", b"").replace(b"\x00", b"") out_file.write(chunk)
18:41
18:41
same as @kanon ig
18:45
btw if it's any helpful, you can also download megabytes, gb and etc
18:45
just instead of bytes write gigabytes
Avatar
unpickled admin bot 07/01/2023 6:52 PM
alr guys clearly the plan is to gget 10 exabytes of storage /j (edited)
Avatar
i'm so mad rn lol
18:54
arx carried them for web for this ctf
18:54
if they didn't have him no web challs
Avatar
unpickled admin bot 07/01/2023 6:54 PM
arx is extremely
18:54
extremely smart
Avatar
unpickled admin bot 07/01/2023 6:54 PM
did you see his xslt chall?
Avatar
read some of his writeups
Avatar
Avatar
unpickled admin bot
did you see his xslt chall?
the one that had 0 solves?
Avatar
Avatar
DreyAnd
the one that had 0 solves?
unpickled admin bot 07/01/2023 6:54 PM
ye
18:54
lol
Avatar
it was on some recent ctf i think
18:56
writeups for the challenges I wrote for dicectf 2023
19:00
around that time i had an XSS via XSLT on a pentest, wish i tried that challenge
19:01
though the idea probably wouldn't cross my mind
19:04
Uncompressed text, on the other hand, will probably still be readable despite the presence of some corrupted bytes. wonder how many bytes we need to get to be able to do this
19:06
alright going to sleep gn guys
19:07
leaving my script overnight maybe it matches something
Avatar
unpickled admin bot 07/02/2023 1:39 AM
weird shit at 8211-8233
01:39
import requests huh = 8211 def pain2(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l + 1}"} ) chunk = response.content return chunk print(pain2(huh, 32))
01:39
(this is the point it switches from a ton of \x00s to a ton of Us)
01:39
b'\x00\x00\x00\x80\xd9\x83\x03\x01\x00\x00\x00\x00 \xff\xd7FPUUUUUUUUUUUUUUUUU'
Avatar
can you type hex
Avatar
unpickled admin bot 07/02/2023 1:40 AM
this is prob compressed flag probably
01:40
uh
Avatar
.hex() but only from 80 to P
Avatar
unpickled admin bot 07/02/2023 1:40 AM
00000080d98303010000000020ffd746505555555555555555555555555555555555
01:40
oh only 80 to
Avatar
its ok
Avatar
unpickled admin bot 07/02/2023 1:40 AM
80d98303010000000020ffd74650
01:43
given the shift from \x00 to U
01:43
and the bytes that are not either
01:43
im pretty confident that is the flag
Avatar
ye i think so
Avatar
unpickled admin bot 07/02/2023 1:44 AM
import requests huh = 8214 def pain2(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk print(pain2(huh, 14)) (edited)
01:44
exact chunk/bytes
Avatar
i dont have pc so cant test
01:44
but you can test decode from example
01:45
if you have any
Avatar
unpickled admin bot 07/02/2023 1:45 AM
uh
01:45
how
Avatar
get a sample with 00 to U
01:46
get the middle hex
Avatar
unpickled admin bot 07/02/2023 1:46 AM
huh
Avatar
and test how to decode it to flag
Avatar
unpickled admin bot 07/02/2023 1:46 AM
wdym
01:46
oh
01:46
uh
Avatar
like play around with it
Avatar
unpickled admin bot 07/02/2023 1:46 AM
dont you need the header
01:46
because of how gzip works?
Avatar
maybe
Avatar
unpickled admin bot 07/02/2023 1:46 AM
and the tail
Avatar
will check tmr
Avatar
Avatar
unpickled admin bot
import requests huh = 8214 def pain2(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk print(pain2(huh, 14)) (edited)
unpickled admin bot 07/02/2023 1:50 AM
anyways this is the exact chunk of interesting bytes
01:50
this pisses me off cuz its so earlyyy
01:50
like
01:50
bruh
Avatar
unpickled admin bot 07/02/2023 1:58 AM
idea also works on 2 (edited)
01:59
(UUU vs \x00 to binsearch)
02:05
second is very weird nvm
Avatar
Avatar
unpickled admin bot
import requests huh = 8214 def pain2(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk print(pain2(huh, 14)) (edited)
unpickled admin bot 07/02/2023 2:06 AM
but anyways if anyone knows how to get flag now that we know where the flag is pls do tell thanks!
Avatar
unpickled admin bot 07/02/2023 2:57 AM
ok ye idk i go sleep now tho
02:57
gl (edited)
Avatar
@unpickled admin bot i think it's the .bin file, not the flag, however that's what we need
05:04
will check
Avatar
@IceCreamMan wants to collaborate 🤝
Avatar
Avatar
unpickled admin bot
80d98303010000000020ffd74650
where did you get the 4650 part from?
05:31
i am thinking, we can recreate a gzip - get a chunk that containers the header + chunk that contains this ^ + chunk with the trailer
05:38
first 50 bytes contain the header + some padding (edited)
05:45
Almost all of the remainder of the file (until the trailer, which begins on byte 4696) is interpreted according to the Huffman tables in figures 8 and 9. As you can see, the Huffman compression tables are very efficient; only 65 bytes of the 4,704-byte gzipped file are "dictionary" information that's needed to interpret the remainder of the file, with an additional 20 bytes of header information and 8 bytes of trailer information. (edited)
Avatar
gunzip -c final.bin [14:53:14] gzip: final.bin: invalid compressed data--format violated
05:53
idk what i'm missing
05:53
import requests huh = 8214 out = open("final.bin", "wb") def header(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def bin(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def trailer(): resp = requests.head("https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz") cl = resp.headers['content-length'] offset = int(cl) - 8 response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{cl}"} ) chunk = response.content return chunk head = header(0, 24) binflag = bin(huh, 14) trail = trailer() out.write(head) out.write(binflag) out.write(trail)
Avatar
Avatar
DreyAnd
gunzip -c final.bin [14:53:14] gzip: final.bin: invalid compressed data--format violated
test similar thing locally first ig
Avatar
Avatar
DreyAnd
where did you get the 4650 part from?
unpickled admin bot 07/02/2023 8:51 AM
binsearched
08:52
o 4650
08:53
wdym 4650
08:53
its the bytes that are there lol
Avatar
they didn't appear for me
Avatar
is anyone able to produce a gz file locally with their header. i.e. use max compression and no filename on top
Avatar
Avatar
unpickled admin bot
b'\x00\x00\x00\x80\xd9\x83\x03\x01\x00\x00\x00\x00 \xff\xd7FPUUUUUUUUUUUUUUUUU'
is there a reason the flag is between 0 and U's? sometimes between U's we can also see similar bytes
Avatar
Avatar
sahuang
is anyone able to produce a gz file locally with their header. i.e. use max compression and no filename on top
i will try in a little
Avatar
a bit different: d88303010000000020ffd74650
Avatar
Avatar
sahuang
is there a reason the flag is between 0 and U's? sometimes between U's we can also see similar bytes
this is how it looks for me
Avatar
d88303010000000020ffd74650 80d98303010000000020ffd74650 only different in first 2 bytes (edited)
Avatar
ok i created example locally with the same header, but no "U"'s, all 00
09:47
hold on
09:47
local test
09:48
also its not the same offset as SEEK offset
Avatar
yeah i have no idea how to decompress the data given the bytes
Avatar
get guesslemonger here 😂
10:46
we're falling
Avatar
Avatar
sahuang
is there a reason the flag is between 0 and U's? sometimes between U's we can also see similar bytes
unpickled admin bot 07/02/2023 12:48 PM
wtf
12:48
huhhhh
12:48
so is that like blocks?
Avatar
no idea
12:49
but looking at chal1 and chal2 file content, i think chal1 is on track, and local test shows there are blocks of 0s and another char (0xaa in my case)
Avatar
unpickled admin bot 07/02/2023 12:49 PM
chal2 is weird
12:49
there are bytes in blocks
12:49
chal1 cant we locally recreate the gzip file, but smaller?
Avatar
yeah that somehow shows chal1 is this stuff
12:49
yeah i did
12:49
also its like 00s aas 00s
12:49
so switch 2 times
12:50
i think same for web server file
12:50
the ending is UUU...00
12:50
so maybe we can ignore middle part and concat or whatever
12:50
didnt try it
12:50
locally i see same pattern
Avatar
unpickled admin bot 07/02/2023 12:51 PM
what if we create a new file
12:52
where we splice together some \x00s, Us, and the flag content
12:52
then try to unzip?
Avatar
sth like that
Avatar
unpickled admin bot 07/02/2023 12:52 PM
actually nvm that prob wont work, theres a crc check
12:52
python gzip may ignore the crc check tho
Avatar
yeah i didnt try
Avatar
unpickled admin bot 07/02/2023 12:53 PM
12:53
ok so
12:53
we need to modify the trailer
Avatar
in fact you can test locally
12:53
yeah
12:53
when testing locally, you need to use extra flag to make header match the web server file
Avatar
unpickled admin bot 07/02/2023 12:53 PM
or just use 4 gigabytes ig
Avatar
is there really a crc check? i was reading RFC it mentioned if there is FNAME(or ftext cant remember) flag set there's no cfc check
Avatar
gzip --n -9 or something
Avatar
and the flag part of header is just null byte, so yeah
Avatar
Avatar
DreyAnd
is there really a crc check? i was reading RFC it mentioned if there is FNAME(or ftext cant remember) flag set there's no cfc check
unpickled admin bot 07/02/2023 12:54 PM
it would make sense
12:54
because
12:54
the trailer is 8 bytes
Avatar
yeah idk why trailer is 8 bytes
Avatar
unpickled admin bot 07/02/2023 12:54 PM
100 bytes
Avatar
null bytes*
Avatar
Avatar
unpickled admin bot
Click to see attachment 🖼️
unpickled admin bot 07/02/2023 12:54 PM
(xxd it)
12:54
one is not null
12:55
hence thats prob the crc?
12:56
4 bytes crc 4 bytes null
Avatar
could be
12:56
is that why i get a format error?
Avatar
unpickled admin bot 07/02/2023 12:57 PM
uh
12:57
probably not
12:57
the magic bytes are different
12:57
101 bytes
12:57
1f8b08 is expected
12:57
not
12:57
8b08
12:58
nvm
12:58
it does start with 1f8b
Avatar
just get a match locally
Avatar
unpickled admin bot 07/02/2023 12:58 PM
huhhhhhhh
Avatar
and test locally
Avatar
unpickled admin bot 07/02/2023 12:58 PM
so idk why format error is happening (edited)
Avatar
Avatar
sahuang
and test locally
unpickled admin bot 07/02/2023 12:58 PM
ye i just have those files from earlier
12:59
you should see the alternating between 0 and another char (edited)
Avatar
unpickled admin bot 07/02/2023 12:59 PM
at the location of the flag?
12:59
thats how it was locally for me
Avatar
not same offset but close to it, also it alternates back later
Avatar
Avatar
unpickled admin bot
so idk why format error is happening (edited)
probably trailer, since mine is all null bytes
13:00
idk much about trailer format
Avatar
Avatar
DreyAnd
probably trailer, since mine is all null bytes
unpickled admin bot 07/02/2023 1:00 PM
<4 bytes crc><4 bytes length of uncompressed data % 2**32> (edited)
Avatar
that's it then
Avatar
Avatar
DreyAnd
import requests huh = 8214 out = open("final.bin", "wb") def header(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def bin(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def trailer(): resp = requests.head("https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz") cl = resp.headers['content-length'] offset = int(cl) - 8 response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{cl}"} ) chunk = response.content return chunk head = header(0, 24) binflag = bin(huh, 14) trail = trailer() out.write(head) out.write(binflag) out.write(trail)
fix the trailer here and should work
Avatar
unpickled admin bot 07/02/2023 1:01 PM
import contextlib @contextlib.contextmanager def patch_gzip_for_partial(): """ Context manager that replaces gzip.GzipFile._read_eof with a no-op. This is useful when decompressing partial files, something that won't work if GzipFile does it's checksum comparison. """ _read_eof = gzip.GzipFile._read_eof gzip.GzipFile._read_eof = lambda *args, **kwargs: None yield gzip.GzipFile._read_eof = _read_eof
13:01
stole thisfrom:
13:01
So here's the problem. I have sample.gz file which is roughly 60KB in size. I want to decompress the first 2000 bytes of this file. I am running into CRC check failed error, I guess because the gzi...
13:01
that way the crc wont annoy us
13:02
that might be it
Avatar
unpickled admin bot 07/02/2023 1:02 PM
from cStringIO import StringIO with patch_gzip_for_partial(): decompressed = gzip.GzipFile(StringIO(compressed)).read() example usage in there
Avatar
give it a try pls
Avatar
unpickled admin bot 07/02/2023 1:02 PM
kk
Avatar
im not close to pc rn
Avatar
unpickled admin bot 07/02/2023 1:03 PM
actually rq
13:03
how bad would it be
13:03
to get the first 10k bytes
13:03
prob not too bad
Avatar
unpickled admin bot 07/02/2023 1:04 PM
just cuz in here hes doign the same thing as us
13:04
kinda
13:04
decompressing first 2000 bytes, for us 10k bytes
Avatar
Avatar
unpickled admin bot
import contextlib @contextlib.contextmanager def patch_gzip_for_partial(): """ Context manager that replaces gzip.GzipFile._read_eof with a no-op. This is useful when decompressing partial files, something that won't work if GzipFile does it's checksum comparison. """ _read_eof = gzip.GzipFile._read_eof gzip.GzipFile._read_eof = lambda *args, **kwargs: None yield gzip.GzipFile._read_eof = _read_eof
unpickled admin bot 07/02/2023 1:08 PM
ook this does not work smfh
Avatar
Avatar
unpickled admin bot
im pretty confident that is the flag
are we even sure about this
Avatar
Avatar
sahuang
not same offset but close to it, also it alternates back later
unpickled admin bot 07/02/2023 1:13 PM
.
13:13
matches up locally
Avatar
unpickled admin bot 07/02/2023 1:28 PM
so wtf does this just zip to zeros
Avatar
@Legoclones wants to collaborate 🤝
13:33
@fleming wants to collaborate 🤝
Avatar
unpickled admin bot 07/02/2023 1:36 PM
smthng tells me we might have the wrong idx? (edited)
Avatar
what's idx
Avatar
unpickled admin bot 07/02/2023 1:39 PM
8214
Avatar
unpickled admin bot 07/02/2023 1:39 PM
but that makes no senseee
13:39
cuz for sahuang the switch was close to where he wrote the file
Avatar
not too close
13:40
off by 200 bytes or so
Avatar
i guess its proportionate
Avatar
unpickled admin bot 07/02/2023 1:40 PM
ye but
13:40
i grabbed 10k bytes (edited)
13:40
shouldve been flag
Avatar
for me (i did 10 mb file) it was really close, for sahuang bit more with gigabytes
13:40
now for exabytes idk
13:41
but you might need more
Avatar
unpickled admin bot 07/02/2023 1:42 PM
first 100000 bytes (edited)
13:42
should i just pull a gigabyte
13:44
did this work locally
13:44
should i be testing this
Avatar
Avatar
unpickled admin bot
should i just pull a gigabyte
go for it
Avatar
unpickled admin bot 07/02/2023 1:45 PM
imma test this
13:45
locally
13:45
the gztool
👍 1
13:50
13:50
sample.gz is my test
Avatar
@afterworld wants to collaborate 🤝
Avatar
maybe try zgrep to see if it'd find the flag format in what you've downloaded so far
Avatar
unpickled admin bot 07/02/2023 1:52 PM
Avatar
Avatar
fleming
maybe try zgrep to see if it'd find the flag format in what you've downloaded so far
unpickled admin bot 07/02/2023 1:52 PM
uhhhh i alr did xxd, and we alr know the bytes where the flag shouyld be
Avatar
Avatar
fleming
maybe try zgrep to see if it'd find the flag format in what you've downloaded so far
flag is written to haystack.bin, which is then gzip'd and its not in flag format
13:52
its compressed
Avatar
gotcha
13:53
zgrep apparently can search for strings in compressed files
Avatar
like search plaintext in compressed file which has random bytes?
Avatar
Zgrep invokes grep on compressed or gzipped files. All options specified are passed directly to grep. If no file is specified, then the standard input is ...
Avatar
also, try to decompress with zcat instead, it's more tolerant @unpickled admin bot
Avatar
unpickled admin bot 07/02/2023 1:54 PM
ok
Avatar
@unpickled admin bot can you try zgrep?
13:54
im out rn, no pc access
Avatar
unpickled admin bot 07/02/2023 1:54 PM
zcat just did gzip -cd
Avatar
unpickled admin bot 07/02/2023 1:55 PM
same with zgrep
Avatar
whats sample.gz
Avatar
unpickled admin bot 07/02/2023 1:55 PM
(I is part of my local testing flag)
Avatar
unpickled admin bot 07/02/2023 1:56 PM
oh fuck my local testing flag is bad
13:56
uh
13:56
welp
Avatar
uiuctf{xxx} just use this?
13:56
then grep uiuc
Avatar
unpickled admin bot 07/02/2023 2:00 PM
14:00
ok zgrep seems to work
14:00
how about their gz
Avatar
unpickled admin bot 07/02/2023 2:00 PM
14:00
same with this (to just unzip)
Avatar
issue is their gz is not complete
Avatar
Okay looks like u guys are doing exactly what I would've
Avatar
so idk if zgrep can catch it
Avatar
you could try truncating the local test gz and see if zgrep still works
Avatar
Avatar
unpickled admin bot
Click to see attachment 🖼️
unpickled admin bot 07/02/2023 2:02 PM
sample.gz is trucated
14:02
i cut off the last 100 bytes
Avatar
You found where bytes changed, extracted, and are trying to decompress
Avatar
unpickled admin bot 07/02/2023 2:04 PM
14:04
:/
14:04
14:05
so what if its not in the first 10k bytes
Avatar
then go for a whole gig
14:05
we can only hope it's somewhere at the top
Avatar
i think i tried the first 200k bytes and it wasnt there
14:06
couldve done it wrong tho
Avatar
zgrep?
Avatar
Avatar
sahuang
zgrep?
unpickled admin bot 07/02/2023 2:07 PM
zgrep finds it cuz it runs on gzip -cd (edited)
14:07
how this gets 20+ solves, so weird
Avatar
maybe we are missing something trivial
Avatar
unpickled admin bot 07/02/2023 2:10 PM
curl so slow :<
Avatar
lol its the web server
14:10
only 4kb/s
Avatar
unpickled admin bot 07/02/2023 2:11 PM
uhuhuhuhuhuhuh
14:11
so can i not try a full gigabyte?
Avatar
yeah idk
Avatar
unpickled admin bot 07/02/2023 2:11 PM
ctf might end b4 that finishes
Avatar
idk if itd be that far in the file
14:12
if it is, this is actually a dogshit tier challenge lmao
Avatar
pretty sure its the front
Avatar
unpickled admin bot 07/02/2023 2:12 PM
should i just
14:12
ctrl c
Avatar
just had to play around with gzip probably
Avatar
unpickled admin bot 07/02/2023 2:12 PM
see if its in here (edited)
14:12
im at 1415k rn
Avatar
you can get 100k bytes fast
Avatar
unpickled admin bot 07/02/2023 2:13 PM
(ctrl c should leave the 1415k bytes in the file) (edited)
Avatar
oh 1415k
14:13
yeah lets do that and pray
Avatar
couldnt you check it while curl is still downloading
Avatar
have you got the last 10k bytes or so@unpickled admin bot
14:13
there is a shift back in last few bytes from U to 0
Avatar
Avatar
jayden
couldnt you check it while curl is still downloading
unpickled admin bot 07/02/2023 2:14 PM
yes
14:14
but im an idiot
Avatar
my assumption was we remove all middle parts and concat them try something
Avatar
unpickled admin bot 07/02/2023 2:14 PM
and i didnt do it
Avatar
Avatar
sahuang
have you got the last 10k bytes or so@unpickled admin bot
unpickled admin bot 07/02/2023 2:14 PM
o
Avatar
you know the total size
Avatar
unpickled admin bot 07/02/2023 2:14 PM
just gonna do this rq
Avatar
just get last 1k or so
Avatar
yeah subtract 1k from the content-length
14:16
and that's the range u need
Avatar
unpickled admin bot 07/02/2023 2:18 PM
bytes=9223372036854774807-9223372036854775807? (edited)
14:18
nothing here
Avatar
Avatar
DreyAnd
import requests huh = 8214 out = open("final.bin", "wb") def header(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def bin(offset, l): response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{offset + l - 1}"} ) chunk = response.content return chunk def trailer(): resp = requests.head("https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz") cl = resp.headers['content-length'] offset = int(cl) - 8 response = requests.get( "https://futuredisk-web.chal.uiuc.tf/haystack.bin.gz", headers={"Range": f"bytes={offset}-{cl}"} ) chunk = response.content return chunk head = header(0, 24) binflag = bin(huh, 14) trail = trailer() out.write(head) out.write(binflag) out.write(trail)
change trailer function here, just do - 1000
Avatar
unpickled admin bot 07/02/2023 2:19 PM
also did 10k (edited)
14:19
wait but thats like the same thing
14:19
content length is 9223372036854775807
14:19
and i did 9223372036854775807-10k
👍 1
Avatar
unpickled admin bot 07/02/2023 2:28 PM
asudhcuiasduchasidcadcadcaaaaa
14:28
so uh
14:28
is it possible we have the wrong index
15:06
are we sure we aren't missing something trivial yet again
Avatar
unpickled admin bot 07/02/2023 3:18 PM
is there an easy way to parse a 10 exabyte file
15:35
we're deffo missing something lol
15:35
how do i time travel guys
Avatar
@Piers wants to collaborate 🤝
Avatar
unpickled admin bot 07/02/2023 4:41 PM
hi piers!
Avatar
Avatar
unpickled admin bot
hi piers!
im here to watch the suffering unfold
🤣 1
💀 2
Exported 618 message(s)